RESTORATION OF IMAGES CORRUPTED BY IMPULSE NOISE 
AND MIXED GAUSSIAN IMPULSE NOISE USING BLIND 

INPAINTING 

MING YAN* 

Abstract. This article studies the problem of image restoration of observed images corrupted by 
impulse noise and mixed Gaussian impulse noise. Since the pixels damaged by impulse noise contain 
no information about the true image, how to find this set correctly is a very important problem. 
We propose two methods based on blind inpainting and Iq minimization that can simultaneously 
find the damaged pixels and restore the image. By iteratively restoring the image and updating the 
set of damaged pixels, these methods have better performance than other methods, as shown in the 
experiments. In addition, we provide convergence analysis for these methods, these algorithms will 
converge to coordinatewise minimum points. In addition, they will converge to local minimum points 
$-H (or with probability one) with some modifications in the algorithms. 
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I— I 1. Introduction. Observed images are often corrupted by impulse noise during 

\^ image acquisition and transmission, caused by malfunctioning pixels in camera sen- 

sors, faulty memory locations in hardware, or bit errors in transmission [T]. There 
are two common types of impulse noise: salt-and-pepper impulse noise and random- 
valued impulse noise. Assume that the dynamic range of an image is [(i„i„, d„ax]- For 
^ images corrupted by salt-and-pepper impulse noise, the noisy pixels can take only two 

^ values d„i„ and d^^^, while for images corrupted by random- valued impulse noise, the 

noisy pixels can take any random value between (i„,i„ and d,„a^. 
I In this work, the original unknown M x N image u is defined on a domain J7 = 

J> {{i,j):i = l,..., M, j = 1, . . . , N}, and the observed M x N image / is modeled as 

00 
O _ J (i?u),j + (ni),j, {i,j)eni, , , 



"^ Here, n2 is the impulse noise, and rii is the additive zero-mean Gaussian white noise. 

^^ H is the identity or a blurring operator, which is assumed to be continuous. The 

subset fij of Jl denotes the region where the information of Hu is missing. The 
problem is to find the true image u from observed image / given the operator H. 

If f2J is empty, there is no impulse noise, then we have / = Hu + ni, which is 
an image denoising (and deblurring) problem, and it has been extensively studied 
i^ by both signal processing researchers and mathematicians. If H.l is not empty and 

known, this can be considered as an image inpainting (and deblurring) problem. 

Here, we will consider the last and most difhcult case where ill i^ ^'^^ empty and 
unknown. The challenge of this problem is to restore the lost details, and remove the 
impulse noise simultaneously. If ni = 0, this problem is an impulse noise removal 
(and deblurring) problem and if rti 7^ it becomes a mixed Gaussian impulse noise 
removal (and deblurring) problem. There are already several types of approaches for 
solving these problems. 

The first type of approaches treats 712 as outliers and uses the ii norm in the 
fidelity term to increase the robustness of inpainting to outliers [2J [3J |31 [S] , and the 



* Department of Mathematics, University of California, Los Angeles, CA, 90095 USA. E-mail: 
yanm@math.ucla.edu. 

1 



2 MING YAN 

problem is to solve 

minimized \{Hu)ij — fij\ + XiJ{u), (1-2) 

where J{u) is a regularization on the true image u. There are many candidates for 
the regularization J{u), and some examples are Tikhonov regularization |B], Geman 
and Reynolds' half quadratic variational models [3, Rudin, Osher and Fatemi's total 
variation models [H E] , and framelet based models (TUl E] . This approach does not 
need to find the damaged pixels and performs well in impulse noise removal. However, 
for the case of images corrupted by mixed Gaussian impulse noise, the Gaussian noise 
is not treated properly. 

The second type of approaches is the two-stage approach [T^[01[TH[TS1[TT1[TB1[T7] . 
which estimates the inpainting region fif before estimating u. In these approaches, 
the second stage becomes a regular image inpainting (and deblurring) problem [l8l 
[1911201 EH 

minimize - ^ ((-ffw)ij - fijf + Ai J(w). (1.3) 

The success of these two-stage approaches relies on the accurate detection of J7f, 
e.g. adaptive median filter (AMF) [22^ is used to detect salt-and-pepper impulse 
noise, while adaptive center-weighted median filter (ACWMF) |23j and rank-ordered 
logarithmic difference (ROLD) [53] are utilized to detect random-valued impulse noise. 
Though adaptive median filter can detect most pixels damaged by salt-and-pepper 
impulse noise, it is more difficult to detect pixels corrupted by random-valued impulse 
noise than salt-and-pepper impulse noise. Recently, by considering two different types 
of noise, Dong et al. |25| proposed a new method using framelet to remove random- 
valued impulse noise plus Gaussian noise by solving 

minimize - ^((iJu),^ + w.^ - f.^jf + Ai||W^u||i + A2 ^ \vij, (1.4) 

where 14^ is a transformation from the image to the framelet coefficients. Two un- 
knowns u (restored image) and v (noise) are introduced into this variational model, 
and their methods can simultaneously find u and v using split Bregman iterations 
[26]. 

Dong et al.'s method uses £1 norm as a convex approximation of £9 term to 
make the result v sparse, and keep the problem convex in the meantime. However, 
using non-convex optimization {(.p when p < \) has better performance than convex 
optimization in dealing with the sparsity, as shown in compressive sensing [27j . Even 
£0 minimization and smoothed t^ minimization are used in many algorithms |28[ I29| 
l30 [ l3 H l32 l l33j. In this paper, we will use (.q minimization instead of ^1 minimization in 
the problem, and by using ^0 minimization, the problem of finding u can be solved by 
considering a problem of finding u and Jli . In addition, using alternating minimization 
algorithm, it can be solved easily by alternately solving the image inpainting problem 
and finding the damaged pixels. 

The work is organized as follows. In section [2] and ^ we introduce our general 
methods for removing impulse noise using two different treatments for the (.q term: I) 
the ^0 term is put in the objective function, II) the ^0 term is in the constraint. The 
algorithms for these two models are similar. The convergence analysis of these two 
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algorithms is shown in section |4] These algorithms will converge to coordinatewise 
minimum points. In addition, they will converge to local minimum points (or with 
probability one) with some modifications in the algorithms. Some experiments are 
given in section [slto show the efBciency of the proposed methods for removing impulse 
noise and mixed Gaussian impulse noise. We will end this work by a short conclusion 
section. 

2. Blind Inpainting Models using £q Term. 

2.1. Formulation. For an M x A^ image, A S {0,1}^^^^ is a binary matrix 
representing a subset fig of the pixels as follows: 

A,, = |l' ifpixel(z,j)ef]„ 
'-' [0, otherwise. 

The connection between binary matrix A and subset fls will be used many times in 
the follow. 

Given a degraded image /, our objective is to estimate the damaged (or missing) 
pixels and restore them. We propose the following model using Iq minimization to 
solve this problem: 



mmimize 



FP{u, v)=\ Y.^{Hu),^, + iH^j - .U^jf + XiJ{u) + \2\\v\\a, (2.2) 



where J{u) is the regularization term on the image, Ai and A2 are two positive param- 
eters. Here P means that i?o term is used as a penalty term in the objective function. 
The parameter Ai is dependent on the noise level of ni. The higher the noise level, 
the larger the parameter should be. The parameter A2 is dependent on the noise level 
of impulse noise. The difference from Dong et al.'s method is that ti norm is replaced 
by (,Q term. It is difhcult to solve this problem because of the €9 term in the function. 
£0 term makes the problem non-convex and the objective function is non-continuous. 



Because what we need to find is just m, we can eliminate v from problem (2.2 1 by 
defining Eq (u) as minF^ (u,v), and the problem becomes 

V 

minimize -Bq (w) = min i^ {u,v). (2.3) 

U V 

However, Eq{u) is still non-convex and difficult to solve. In order to solve this prob- 
lem, we will transform the problem into a continuous and multi-convex problem of u 
and A by introducing a new variable A, and by solving the new problem of u and A, 



we can obtain a local optimal solution for the original problem (2.3) of u only. 

First of all, we provide the intuition behind choosing £q minimization instead of 
£1 minimization for v. Because of the speciality of Iq minimizations, v can be easily 
eliminated from min_F^(u, v) and we can obtain the function of u only as follows: 

V 

<(«) - ^ E M{Hu),,j " kj) + Ai J(zi), (2.4) 

where Ro{x) = min(|a;p, 2A2). Similarly, we can obtain the function of u only when 
£1 term is used instead of £q term as follows: 

^r(") ^ IJ2Ri((Hu),^^ ~ /,,,) + AiJ(u), (2.5) 
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|x|2, if|x|<A2, 

2A2|a;| — A|, otherwise. 
The data fidehty terms ^ Y.i,j Ra{{Hu)^^.j - /»,-,) and \ J2i,j Ri{{Hu)ij - fij) in 
problems (2.4) and (2.5) are used to approximate the negative log-hkeHhood resulting 



where Ri{x) 



from the mixed Gaussian impulse noise model, with each Rq or Ri describing the 
negative log-likelihood for each pixel, because the noise is independently distributed 
at all pixels. What we are trying to do is finding better and simpler model for mixed 
Gaussian impulse noise. We can simulate the probability distribution of the pixel 
values when it is corrupted by both additive Gaussian and random-valued impulse 



noise. For a fixed pixel value (128 in Fig. 2.1 ), a value is added as a Gaussian noise 



and it is replaced by any random value between d^^^ 

with some probability related to the impulse noise level 

and the approximated negative log-likelihood function is shown in Fig. 2.1 



and dniax (0 and 255 in Fig. 2.11 

This is run for 10® times 

We can 



see that it is a constant when the value is far from the true pixel value. In this figure 
we also show Rq{x — 128) and Ri{x — 128) with some scaling and lifting. 



35r 

30 
■a 

§25 1 

~ 20 

en 
o 

CD 15 

> 

■fa 

5 

0, 



negative log-likcliho jd 




50 



100 



150 200 
pixel value 



250 



300 



Fig. 2.1: Negative log- likelihood value for the mixed Gaussian impulse noise, and 
comparison with Rq and Ri 



From Fig. 2.1 we can find that it is more reasonable to use io term instead of £i 
term. However, £i minimization makes the problem convex and it is easier to find the 
solution, while the problem is non-convex and difficult to solve if £q minimization is 
used. Next, we will introduce an auxiliary variable A or 51 1 and we can find a local 
minimizer of Eq (u) by solving the new problem of u and A. 

For any fixed u, we can find the optimal v by solving the following optimization 
problem: 



The solution is 



minimize - \^{Hu + v — fY + A2||w||o- 




{Hi 



"J' 



or /ij - {Hu)tj, 



if I/,,, - (i?z2),,,f > 2A2, 
if I/,,, - (i?M),,,f = 2A2. 
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When Vi,j ^ 0, we have Vij — ftj — {Hu)i,j. Therefore if we denote 

then we have a new problem with u and A as follows: 

minimize^ Fi{u, A) ee - ^ A,j((iJu),j - f,^jf + Ai J(w) + A2 ^(1 - Aij). 

(2.7) 



Problem (2.7) can be solved easily by alternating minimization method, and the 



algorithm for solving (2.71 is described in section 2.2 For a general alternative mini- 
mization procedure for convex and non-convex problems, please see |34] . 

Remark: In fact, the constraint of A e {0, IJ^^^'^ can further be relaxed into 
A e [0, 1]*^^^^ and we have the following multi-convex problem: 

minimize Fi(u,A) = - V A,j((iJu),j - /ij)^ + Ai J(u) + A2 V(l - A,.j). 

(2.8) 

If u is fixed, Fi{u,A) is a function of A only and it is separable. The optimal 
solution for A with fixed u is 





fo. 


if 1/,,, - (ilu),.,f > 2A2 


id ^ 


^' 


if 1/,,, - (ilu),.,f < 2A2 




1 1, 


if 1/,,, - (ilu),.,f = 2A2 



where i is or 1 for unrelaxed problem (|2.7[), and t is any number in [0,1] for 



relaxed problem (2.8). If we eliminate A as before, mm/^fziQijMxN Fi{u,A) and 
min^grQjiA/xJv Fi{u,A) are functions with respect to u only and same as Eq{u). Be- 
cause all the problems are non-convex, there may exist many local optimal solutions 
which m ay no t be global optimal solutions. We will show in section [4] that by solving 



problem (2.8), we can obtain a local minimizer of function Eq{u 



2.2. Algorithm. The objective function defined in (2.8) is non-convex. It is 
still difficult to solve it in the pair (w,A), but we can use alternating minimization 
method, which separates the energy minimization over u and A into two steps. For 
solving the problem in u with A fixed, it is a convex optimization problem for image 
inpainting and the problem of finding A with u fixed can be solved in one step. These 
two subproblems are 

1) Finding u: Given an estimate of the support matrix A, the minimization over 
u is just an image inpainting (and deblurring) problem |35j : 

minimize- ^ ((iJw),^ - /.j)^ + Ai J(m). (2.9) 

(»j)eni 

There are many existing methods for solving this problem. 

2) Finding A: Given an estimate of the image u, the minimization over A becomes: 

minimize - V A,,J{{Hu)^,j - fijf - A2 V Aij. (2.10) 





f ° 


if ((i7u),,j - 


- kjf /2 > \2 


A.,, = 


< or 1 


if {{Hu),^, - 


- k,f /2 = A2 




I 1 


if {(Hu),,, - 


- /.,,)' /2 < A2 
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Since this mininiization problem of A is separable, it can be solved exactly in only 
one step: 



(2.11) 



Therefore, the proposed algorithm for blind inpainting with ^q minimization is it- 
eratively finding u and A. As mentioned in section 12. ll minAg{o,i}J>f>;« -F'i(m, A) = 
minyYg ro^ijMxN Fi(m, A) for all fixed u, thus we can force Ai-, e {0,1} during the 
algorithm. When {{Hu)ij — fij) /2 — A2, we can randomly choose A^ ^ to be or 1. 
The detailed algorithm for blind inpainting is described below, the initial A° is 
chosen by the methods for detecting the impulse noise (AMF for salt-and-pepper im- 
pulse noise and ACWMF for random- valued impulse noise). Usually three iterations 
are sufhcient, as shown in the experiments. 

Algorithm 1 Proposed blind inpainting algorithm. 

Input: /, Ai, A2, A°, e 
Initialization: k — 1. 

while fc < 2 or i^i(w'=,A'=) - Fi{u^~^ , A''-^) > e do 

Obtain u'^ by s olving (2.9). 



Obtain A*^ by ( |2.11[ ). 
k = k + l. 
end while 



Here e is chosen to be small and served as a stopping criteria to stop the algorithm 
when the difference in function values between two iterations is too small. Ai and A2 
are two parameters depending on the noise levels of ni and 712 . 

Remark: This algorithm and the algorithm in next section depend on the ini- 
tial A°, and choosing a better A° will reduce the total number of iterations and the 
restoration result (because of the non-convexity of the problem). Therefore, we can 
choose the result of AMF and ACWMF for salt-and-pepper and random-valued im- 
pulse noise respectively as initial A*'. For salt-and-pepper impulse noise, AAiF will 
provide a very accurate initial guess for A for most cases and the improvement from 
more iterations is not too much, while for random-valued impulse noise, the output 
of ACWMF is not accurate, and more iterations are needed to improve the detection 
of corrupted pixels and the recovery result. 

3. Blind Inpainting Using Adaptive Outlier Pursuit. In the previous sec- 
tion, we proposed a method for blind inpainting by putting a io term in the objective 
function, which can be solved by iteratively updating the set of pixels damaged by 
impulse noise (or the binary matrix A e {0, i].^^><^) and restoring the image. Instead 
of putting the Iq term in the objective function, we can also put a constraint on ||w||o, 
which will be equivalent to a constraint on ^^ A^j. This technique has been applied 
to robust 1-bit compressive sensing where there are sign flips in the binary measure- 
ments belonging to {—1, 1} |36j and robust matrix completion [37J. We proposed an 
algorithm, named adaptive outlier pursuit (AOP), which can adaptively find the sign 
flips (outliers) and reconstruct the signal by using other measurements assumed to be 
correct. Since images corrupted by impulse noise can also be considered as sparsely 
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corrupted measurements, the same idea can be applied in impulse noise (and mixed 
Gaussian impulse noise) removal by iteratively finding the pixels corrupted by impulse 
noise and recovering the image using other pixels. 

Let us assume that the number of pixels corrupted by impulse noise is bounded 
above by a integer L, this can be obtained from the noise level of impulse noise. 
Therefore, the new problem is 

minimize \Y.t.j{{Hu)i.j + Vi,j - fi^.jY + \iJ{u), 

"'" (3.1) 

subject to ||u||o < i, 

which can be written as 

minimize F'^(u,i;) = -;.'^{{Hu)i^j + v^j - fi^jf + \iJ{u) + ii^^,\\^\\^<^L-}, (3.2) 

where '-{i,:||i,||o<l} is the indicator function equals to zero when \\v\\q < L and +oo 
otherwise. Here C means that the £o term is put in the constraint. We can further 
eliminate v, and the problem of u only is: 

minimize i?o (w) = mini^ (m,i;). (3-3) 

U V 

Similarly, we can introduce new variation A and the corresponding problem of u 
and A is 

minimize J^a ^^t.]{{Hu),,j - f.^^.j)'^ + XiJ(u), 

u,A 'J (34) 

subject to Y.i,j{^ ~ Kj) < L, K,j G {0, 1}, 
which can be described in another way as 

mininiize F2(u, A) ee V -A,_j{{Hu),_j - fijf + XiJ{u) + i.{A:J2, ,(i-Ai i)<L}, 

(3.5) 



«j 



where t/A:V;. (i-A; )<l} is the indicator function equals to zero when ^^ (1 — A^j) < 
L and +oo otherwise. Note that we can also relax the constraint A e {0, 1}*^^^ to A € 
[0, 1]^^^^ and the problem becomes a multi-convex problem, as done in the previous 
section. In addition we have miuAg/o ija^xjv _F2(m, A) = minAgrg ijmxn -F2(u, A) = 
E^iu). 

This problem can also be solved iteratively as in the previous section. The u- 
subproblem is same as the previous one, and the A-subproblem is slightly different. 
In order to update A, we have to solve 

minimize J2^,j KAiHu),^^ - /»j)V2, 
subject to X;^j(l - Kj) < L; A^.j G {0, 1}. 

This problem is to choose M x N — L elements with lea st su m from M x N elements 
{{{Hu)ij — fij)'^/2}^J^ ,j^. Given a u estimated from (2.9), we can update A in one 
step: 

_rO, if((iI«),,,-/,,,)V2>A2, , . 
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where A2 is the L*'' largest term of {{{Hu),^j - Aj)^/'^}f^uj=i- I^ the L*'' and 
(L + 1)*'' largest terms are equal, then we can choose any binary matrix A such that 
J2^j ^i.j ^M xN-L and 

min i{Hu),,j - Uj)y2 > max {{Hu\, - h,,f /2. (3.8) 

The algorithm for blind inpainting using AOP is described below. 

Algorithm 2 Proposed blind inpainting using AOP. 

Input: /, Ai, L, A", e 
Initialization: fc = 1. 

while fc < 2 or F2(u'' , K^) ~ F2(u^-'^ , K''-^) > e do 

Obtain u*^ by solving (2.9). 



Obtain A'^' by ( |3.7[ ) 
k = k + l. 
end while 



Here e, A° and Ai are chosen in the same way as algorithm[l] The integer L is an 
estimation of the number of corrupted pixels, which is easy to obtain from the noise 
level of the impulse noise. 

The difference between these two algorithms is the A-subproblem, the threshold 
A2 is fixed for algorithm [T] while A2 is changing for AOP. AOP can be considered as 
one special case of algorithm [T] with changing A2. However the performance of these 
two algorithms is similar, and the parameter L is easier to obtain than A2. So we will 
only use algorithm [2] for numerical experiments. 

4. Convergence Analysis. In this section, we establish some convergence re- 
sults for these two algorithms. We will show that these two algorithms will stop 
in finite steps, and the output is a coordinatewise minimum point of Fi(u,A) (or 
F2{u,A)) with relaxed constraint A^j G [0,1]. A point (it, A) is a coordinatewise 
minimum point of Fi{u, A) (or F2{u, A)) means that u is a minimizer of Fi{u, A) (or 
F2(u, A)) and A is a minimizer of Fi{u,A) (or F2{u,A)). In addition, we can modify 
a little bit in the algorithm and the output will be a local minimizer of Eq{u) (or 
Eq{u)) (or with probability one). 

Since the convergence analysis is similar for both algorithms, let F(u, A) stand 



for Fi(u, A) in the penalty problem (2.7| and F2(u, A) in the constraint problem (3.5), 
and Eq{u) stand for Eq{u) and Eq{u) as uim/^^tQ ijmxn F{u,A). 

Before deriving the convergence results of the algorithms, two theorems are intro- 



duced to show that we can solve problems (2.7) and (3.5) of u and A to find a local 
minimizer for Eq{u). 

Theorem 4.1. Ifu* is a local minimizer of Eo{u), then for any A* G {0, l}*^^^ 
minimizing F{u*,A), (u*. A*) is a local minimizer of F{u, A). 

Proof. Since u* is a local minimizer of Eo{u), we can find e > such that for all 
u satisfying ||u — m*|| < e, we have Eq{u) > Eo{u*). Therefore for all (w. A) satisfying 
II (m. A) - (m*,A*)|| < e, we have ||m-m*|| < e. 

F{u,A) > Eo{u) > Eo{u*) - F{u*,A*). 

Thus {u* , A*) is a local minimizer of F{u, A). D 
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To obtain a sufficient condition for u* being a local niinimizer of Eq{u), we have 
the following theorem. 

Theorem 4.2. Given fixed u* , if for all A € {0,1}*^^^ minimizing F{u*,A), 
we also have that u* minimizing F(u,A), then u* is a local niinimizer of Eq{u). 



Proof. We will prove it for the penalty problem ( 2.7 1 with F{u, A) = Fi (u, A) first. 
Let n+ = {(i, j) : |/,^^-(iJu*),,,f > 2A2} and ft. = {{i,j) : \f^^j~{Hu*),J^ < 2A2}, 



then from the continuity oi H, we can find e > such that when ||m — u*|| < e, we have 

fij ~ {Hu)ij\'^ > 2A2 for all {i,j) G n+ and \fij - iHu)ij\ 



l/jj - (-ffu)ijP > 2A2 for all {i,j) e n+ and |/jj - (-ffu)ijp < 2A2 for all {i,j) G f7_ 



Notice that Eq{u) = Tainf^^^Qi^MxN F{u,A), then there exists A G {0, IJ^^^ such 
that Eo{u) = F{u,A). We have A^., — when {i,j) G ft+ and Aij = 1 when 
{i,j) G r2_. Thus A is also a minimizer of Fi{u*,A). In addition, we have Eo{u*) = 
F{u*,A) < F{u,A) = F{u). Therefore, u* is a local minimizer of Eq{u). 



For the constraint problem (3.5) with F{u,A) = F2(u, A), we have to just replace 



A2 with the L*'* largest term of {{{Hu)ij — /i,j)^}j^\ j^i, and the result follows. D 

Remark: We can replace the {0, l}*^><^ with the relaxed version [0, 1]^^^^ and 
both theorems are still valid. 

With these theorems, we are ready to shown the convergence results of the two 
algorithms. 

Theorem 4.3. Both algorithms will converge in finite steps and the output 
(u*. A*) is a coordinatewise minimum point of F{u, A). 

Proof. As explained before, though we have Aij e [0, 1] for the relaxed problem, 
we can always force A^j- G {0, 1} during the iterations because for every fixed u, we 
can also find a minimizer of F{u,A) satisfying A^ ^ G {0,1}. Since Aij G {0,1}, 
there are only finite number of A's (the total number of different A's with constraint 
Aij G {0, 1} is 2*^^^) and the algorithm will stop in finite steps if the u-subproblem 
and A-subproblem are solved exactly. Assume that at step fc, the function F{u, A) 
stops decreasing, which means 

F(u^ A'=) = F(u'=+\ A'^+i). (4.1) 

Together with nonincreasing property of the algorithm 

F(M^A'^■) > F{u''+\A'') > F{u''+\A''+^), (4.2) 

we have 

F(M^ A*--) = F(u'=+\ A*--) = F(m'=+\ A'^+i). (4.3) 

F(u^ A'^) ^ F{u''+\A'') = minF(u, A*^), (4.4) 

U 

F(u^ A*^) = minF(u''', A). (4.5) 

Then (u*. A*) = (u*^, A*^) is a coordinatewise minimum point of F{u,A). D 

However, coordinatewise minimum point (u. A) may not be a local minimum 

point of F{u, A). As shown in the next theorem, A being the unique minimum point 

of F{ii, A) is a sufficient condition for ({t. A) to be a local minimum point. 

Theorem 4.4. For a coordinatewise minimum point (u*,A*) of F{u, A), if A* 

is the unique minimum point of F{u*,A), then (u*. A*) is a local minimum point of 

F{u,A). Furthermore, u* is a local minimum point of Eq{u). 



Thus 
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Proof. Since A* is the unique minimum point of F{u*,A) and u* minimizes 



-F(u, A*), we have u* being a local minimum point of Eq{u) from theorem 4.2 Then 



(m*, A*) being a local minimum point of F{u,A) follows from theorem 4.1 D 



Let (m*, A*) be a coordinatewise minimum point of F{u,A). From theorem 4.4 



if u* is not a local minimum point of Eo{u) , then there are many minimum points for 



F{u*,A). In addition, from theorem 4.2 there exists another minimum point A of 
F(u*,A), such that F(u*,A) > min„F(u, A). 

Based on theorems |4.2| and |4.4[ we have the following two corollaries. 

Corollary 4.5. When solving the subproblem of finding A'', if there are many 
minimum, points for F{u^,A), we can choose the best A with lowest mmu F{u, A), 
then the algorithm will stop at a local minimum point of Eq{u). 

Proof. When the algorithm stops at (u*,A*), we have F{u*,A) = min,i F(u, A) 
for all A £ {0, 1}*^^^ minimizing F{u*,A), otherwise, we can find a A giving lower 
min„ F{u, A) and the algorithm continues. Then from theorem |4. 2 [ we know that u* 
is a local minimum point of Eq[u). D 

Remark: This strategy for choosing A may not be very useful for practical 
computation because there could be a large number of candidates for such A. However, 
in the numerical experiments, this does not happen, because of the rounding error in 
the calculation and the probability for two values to equal is 0. 

Instead of having to choose the best candidates from many candidates, we can 
modify the function F{u,A) to avoid this case, as Wang et al. did in [55] . 

Corollary 4.6. // we can modify the objective function F{u,A) by adding 
T^- ■ Aij-Tij, where {ri_j} are random values uniformly distributed in [0,1] and r 
is a small number, then the algorithm will stop at a local minimum of Eq{u) = 
min^g r-Q ]^iMxiv(i^(M, A) + t^^ ■ Aijrij) with probability one. 

Proof. In this case, the subproblem for updating A is 

if ((Hw),j-/,.,)V2 + Tr,,j>A2, 
A,j = <( or 1 if ((Hw).j - /,.,)' /2 + rr,^, = A2, (4.6) 

if ((i/7/),,, -/,j)V2 + Tr,j <A2. 




The probability of getting {{Hu)ij — fij) /2 + Tri_j = A2 is because of the ran- 
domness of rij. Then the algorithm will converge to a local minimum of Eq{u) with 
probability one. Similarly for AOP, the probability of L*'* and (L + 1)*'' largest term 
being equal is 0. D 

Remark: Besides adding additional term onto F{u, A) we can also add a small 
random variable rAijrij onto fij. 

5. Numerical Experiments. Because the performance of these two algorithms 
is similar and it is easier to obtain an approximate value for the number of damaged 
pixels, in this section, we apply the blind inpainting algorithm using AOP to remove 
impulse noise and mixed Gaussian impulse noise. 

As mentioned in the introduction, there are many different choices for J{x), and 
the performance of this algorithm depends on which J{x) is chosen. In order to make 
the comparison with other methods fair, we choose J{x) to be total variation for all 
methods used in the numerical experiments. Therefore the optimization problem will 
be different from the ones that proposed in the literature. For example, we implement 
the algorithm of paper [25. by replacing the wavelet frame with total variation |39| . 

If the regularization for image u is total variation, and H is the identity operator. 
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the step for finding u is 

minimize y^ i^^iji^ij - fi.jf + AiTV(u), (5.1) 

which is the famous TV inpainting model |40[ I41j . Numerous algorithms proposed for 
solving TV denoising problem can be adopted to solve this TV inpainting problem 
with some necessary modifications. Some examples are algorithms based on duality 
[321133], augmented Lagrangian methods [331133, and split Bregman iterations |^H5] . 
In the numerical experiments, we choose the split Bregman iteration to solve the 
subproblem. 

To evaluate the quality of the restoration results, peak signal to noise ratio 
(PSNR) is employed. Given an image u £ [0,255]^^''^, the PSNR of the restora- 
tion result u is defined as follows: 

PSNR(u, u) = 10 logio ^ . (5.2) 

There are two important types of impulse noise: salt-and-pepper impulse noise 
and random-valued impulse noise. The pixels damaged by salt-and-pepper impulse 
noise are much easier to find since the values are either (i,ni„ or d^^^. The adaptive 
median filter (AMF) has been widely used to accurately identify most pixels dam- 
aged by salt-and-pepper impulse noise (See e.g. |47l |22]). The detection of pixels 
corrupted by random-valued impulse noise is much harder than salt-and-pepper im- 
pulse noise because the value of damaged pixels can be any number between d,-„i„ and 
d„ax- AC WMF was proposed to detect pixels damaged by random- valued impulse 
noise. 

For the first experiment, salt-and-pepper impulse noise is considered. Because the 
pixels corrupted by this kind of impulse noise can only take two values, the detection 
of damaged pixels is easy. As an efficient method for detecting the damaged pixels, 
AMF is used widely in salt-and-pepper impulse noise removal. We will compare total 
variation blind inpainting using AOP with AMF and TVLl, where TVLl is the result 
of solving the following problem. 






minimize^ \ui_j — fi_j\ + X TY{u), (5.3) 



using split Bregman [46] . The parameter A is tuned to achieve the best quality of the 
restoration images. 

Four test images are corrupted by Gaussian noise of zero mean and standard 
deviations a = 5, 10, 15, then we add salt-and-pepper impulse noise with different 
levels (s = 30%, 50%, 70%) on the test images, with or without the Gaussian noise. 
The PSNR values of the results from three methods are summarized in Table \5A\ 

From Table |5.1[ we can see that for salt-and-pepper impulse noise, the results 
from total variation blind inpainting using AOP are better than those by AMF and 



TVLl for all noise levels. The visual comparison of some results is shown in Fig. 5.1 
We can see noisy artifacts in the background of the images obtained by AMF, and the 
images obtained by TVLl are blurred with some lost details. Images restored by total 
variation blind inpainting using AOP are smooth in fiat regions of the background 
and the details are kept. 
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Salt-and-Pepper Impulse Noise | 


a + s 


"Lena" 


"House" 1 


Noisy 


AMF 


TVLl 


AOP 


Noisy 


AMF 


TVLl 


AOP 


0+30% 


10.68 


33.80 


30.97 


37.75 


10.42 


38.97 


36.53 


47.14 


5+30% 


10.66 


31.47 


30.32 


34.56 


10.40 


33.69 


34.49 


39.09 


10+30% 


10.62 


27.93 


29.40 


32.25 


10.39 


28.90 


32.79 


35.73 


15+30% 


10.54 


25.14 


28.59 


30.41 


10.30 


25.66 


31.41 


33.49 


0+50% 


8.44 


30.35 


27.98 


33.98 


8.22 


34.60 


31.70 


42.50 


5+50% 


8.45 


29.00 


27.58 


32.61 


8.19 


31.73 


31.36 


37.43 


10+50% 


8.42 


26.54 


27.25 


30.88 


8.18 


27.84 


30.37 


34.60 


15+50% 


8.40 


24.15 


26.46 


29.50 


8.14 


24.96 


29.60 


32.50 


0+70% 


7.00 


26.85 


24.90 


30.61 


6.75 


30.05 


26.80 


36.84 


5+70% 


6.97 


26.11 


24.65 


29.94 


6.73 


28.80 


26.81 


34.44 


10+70% 


6.98 


24.62 


24.57 


29.05 


6.74 


26.15 


26.36 


32.28 


15+70% 


6.97 


22.83 


24.20 


28.11 


6.72 


23.86 


25.85 


31.23 




"Cameraman" 


"Boat" 


Noisy 


AMF 


TVLl 


AOP 


Noisy 


AMF 


TVLl 


AOP 


0+30% 


10.32 


33.62 


30.43 


38.43 


10.70 


30.16 


27.64 


33.32 


5+30% 


10.28 


31.34 


30.09 


35.33 


10.69 


30.27 


27.84 


33.39 


10+30% 


10.25 


28.15 


29.35 


32.47 


10.69 


30.34 


27.70 


33.06 


15+30% 


10.21 


25.39 


28.40 


30.45 


10.68 


30.24 


27.70 


32.96 


0+50% 


8.08 


29.78 


26.80 


34.58 


8.49 


27.27 


25.00 


30.54 


5+50% 


8.08 


28.35 


26.55 


32.78 


8.51 


27.20 


25.02 


30.49 


10+50% 


8.09 


26.38 


26.16 


30.57 


8.48 


27.24 


25.24 


30.19 


15+50% 


8.04 


24.29 


26.09 


29.08 


8.49 


27.12 


25.02 


30.12 


0+70% 


6.62 


25.73 


23.20 


29.85 


7.02 


24.33 


22.42 


27.20 


5+70% 


6.62 


25.22 


23.21 


29.06 


7.02 


24.19 


22.35 


27.14 


10+70% 


6.60 


24.02 


23.01 


28.04 


7.02 


24.18 


22.42 


27.08 


15+70% 


6.60 


22.48 


22.66 


26.96 


7.01 


24.23 


22.37 


26.97 



Table 5.1: PSNR(dB) for denoising results of different algorithms for noisy images 
corrupted by salt-and-pepper impulse noise and mixed Gaussian impulse noise, a is 
the standard deviation for the Gaussian noise and s is the level of salt-and-pepper 
impulse noise. 



We do not compare AOP with two-stage approaches because the detection of 
damaged pixels by salt-and-pepper impulse noise using AMF is very accurate, and 
A will not change too much during the iterations, thus the performance of AOP will 
be similar to the two-stage approach by first detecting the damaged pixels by AMF 
and then solve the total variation image inpainting problem, which is just the first 
iteration of AOP. Therefore, for the cases with salt-and-pepper impulse noise, using 
blind inpainting does not improve too much by adaptively updating A, and our focus 
will be on random- valued impulse noise. 

For random-valued impulse noise removal, it is more difficult to detect the cor- 
rupted pixels because they can take any value between dmin and dmax ■ ACWMF and 
ROLD are used in two-stage approaches for detecting the damaged pixels (531 Ell- 
in this experiment, we will compare AOP with ACWMF, TVLl, and two-stage ap- 
proaches for random-valued impulse noise removal. The two-stage approach we used 
here is just one step of AOP, and the parameter for second stage (total variation 
image inpainting) is also tuned to achieve the best quality of the restoration images. 
In addition, we will compare with two other methods, one is Dong et al.'s method [2 5) 
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Fig. 5.1: Denoising results of images contaminated by both Gaussian noise and salt- 
and-pepper impulse noise with cr = 10 and s = 30%. Top row: noisy images; Second 
row: the results restored by AMF; Third row: the results restored by TVLl; Bottom 
row: the results restored by total variation blind inpainting using AOP. 



for solving problem 



minimize - > {{Hi 

1» 7) V ' ^ 



kjf + XiTV{u) + X2j2K 



(5.4) 



»j 



»j 



The parameters Ai and A2 are chosen to achieve the best quality. The other is the 
penalty decomposition method (PD) by Lu and Zhang [33 for the problem 



minimize i J^i i{{Hu)^,j + Vi,j - f^,,)'^ + XiTV{u), 

U,V '^ u 

subject to ||i;||o < L. 



(5.5) 



Ai is tuned to achieve the best quality of the restoration images. 

Again four test images are corrupted by Gaussian noise of zero mean and standard 
deviation {a = 10, 25), then we add random- valued impulse noise with different levels 
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[s = 25%, 40%) onto the test images, with or without Gaussian noise. The PSNR 
values ol the results from these six methods are summarized in Table 15.21 







Random- Valued 


Impulse 


Noise 










a + s 


Noisy 


ACWMF 


TVLl 


Two- St age 


[25. 


PD 


AOP 


"Lena" 


0+25% 


15.25 


30.53 


31.75 


32.65 


31.09 


33.42 


33.74 


10+25% 


15.26 


30.44 


31.87 


32.65 


31.21 


33.42 


33.66 


25+25% 


15.11 


29.23 


30.37 


31.32 


30.13 


32.36 


32.64 


0+40% 


13.27 


24.62 


29.22 


28.44 


28.61 


30.39 


30.77 


10+40% 


13.22 


24.31 


28.94 


28.15 


28.41 


30.14 


30.34 


25+40% 


13.05 


23.42 


28.02 


27.23 


27.40 


29.23 


29.54 


"House" 


0+25% 


14.71 


31.50 


36.88 


35.55 


35.04 


41.38 


42.11 


10+25% 


14.71 


31.38 


36.36 


35.55 


35.20 


41.36 


41.61 


25+25% 


14.49 


30.48 


35.51 


34.62 


34.33 


40.54 


40.85 


0+40% 


12.65 


23.90 


32.77 


30.07 


31.45 


35.53 


37.39 


10+40% 


12.62 


23.82 


32.44 


29.97 


31.24 


35.24 


36.71 


25+40% 


12.34 


22.93 


31.52 


29.02 


30.38 


34.32 


35.86 


"Cameraman" 


0+25% 


14.48 


28.93 


31.24 


31.75 


30.54 


32.72 


33.16 


10+25% 


14.52 


28.99 


31.32 


31.86 


30.54 


32.72 


33.26 


25+25% 


14.10 


27.98 


30.43 


30.93 


29.63 


31.80 


32.55 


0+40% 


12.37 


22.26 


27.36 


26.51 


27.00 


28.39 


29.16 


10+40% 


12.39 


22.42 


27.75 


26.74 


27.37 


28.42 


29.21 


25+40% 


12.02 


21.32 


26.69 


25.56 


26.30 


27.43 


28.11 


"Boat" 


0+25% 


15.29 


28.18 


28.63 


29.37 


28.26 


29.48 


29.60 


10+15% 


15.35 


28.16 


28.76 


29.48 


28.36 


29.61 


29.78 


25+25% 


14.01 


27.03 


27.73 


28.58 


27.43 


28.62 


28.81 


0+40% 


13.30 


23.56 


26.16 


26.03 


25.88 


26.88 


27.12 


10+40% 


13.31 


23.42 


26.22 


25.96 


25.90 


26.88 


26.99 


25+40% 


13.12 


22.37 


25.27 


25.00 


24.79 


25.93 


26.02 



Table 5.2: PSNR(dB) for denoising results of different algorithms for noisy images 
corrupted by random- valued impulse noise and mixed Gaussian impulse noise, a is 
the standard deviation for the Gaussian noise and s is the level of random-valued 
impulse noise. 



From Table [5T2| we can see that for random-valued impulse noise, the results from 
total variation blind inpainting using AOP are better than those by other methods for 
all noise levels. The comparison of ACWMF and TVLl shows that TVLl outperforms 
ACWMF for all noise levels tested, because ACWMF misses quite a lot of real noise 
and false-hits some noise-free pixels. TVLl has better performance than two-stage 
approach for the cases when noise level is high (s = 40% in the numerical experiments), 
because the accuracy of detecting corrupted pixels by random-valued impulse noise 
using ACWMF is very low when the noise level is high. The accuracy of detecting 
corrupted pixels can be improved by our method via iteratively updating the binary 
matrix A, as shown in the comparison. PD outperforms other methods except AOP 
because Iq term is used in the problem. The problem PD solves is a non-continuous 
problem and it will stop at a local minimum (u* , v*). However, u* may not be a local 
minimum of the problem with u only. While AOP will converge to a local minimum 
of the problem with u only. 



The visual comparison of some results is shown in Fig. 5.2 We can see noisy arti- 
facts in the background of the images obtained by ACWMF, and the images obtained 
by TVLl are blurred with some lost details. Images restored by total variation blind 
inpainting are smooth in flat regions of the background and the details are kept. 

Both experiments show that our method by iteratively updating the inpainting re- 
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Fig. 5.2: Denoising results of images contaminated by both Gaussian noise and 
random-valued impulse noise with cr = 10 and s — 25%. Top row: noisy images; 
Second row: the results restored by ACWMF; Third row: the results restored by 
TVLl; Bottom row: the results restored by total variation blind inpainting using 
AOP. 



gion and performing image inpainting provides better results in identifying the outliers 
and recovering damaged pixels. For salt-and-pepper impulse noise, because there are 
very accurate methods for detecting the corrupted pixels such as AMF, our method 
has similar performance as two-stage approaches. However, for random-valued im- 
pulse noise, there is no method can detect corrupted pixels accurately, especially when 
the noise level is high. Our method by iteratively updating the corrupted pixels is a 
better choice. 

At the end of this section, we compare the damaged pixels detected by ACWMF 
and obtained from AOP in Fig. |5.3| for the cameraman image. The damaged pixels 
are chosen randomly (s = 40%). The pixels with black color are detected as damaged. 
The set obtained from AOP is also random and does not contain any information from 
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Fig. 5.3: The damaged pixels detected by ACWMF (left column) and AOP (right 
column) . 



the image, while the set detected by ACWMF still has some features from cameraman 
image. 

6. Conclusion. This paper presents two general algorithms based on blind in- 
painting and io minimization for removing impulse noise. The difference is in the 
treatment for the £o term: 1) the £q term is put in the objective function, II) the io 
term is in the constraint. Both problems can be solved by iteratively restoring the 
images and identifying the damaged pixels. The performance of these two methods is 
similar, and the connection between these two methods is shown. It is also shown in 
the experiments that the proposed methods perform better than other methods. This 
simple idea can also be applied to other cases where the noise model is not Gaussian. 
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